The classical arts of Japan (pre-1868) have a very different structure from the modern arts. The classical arts are entirely defined by their kata. If you take something like Suio Ryu or Shinto Muso Ryu, they have a clearly defined set of kata. Changing the kata is frowned upon, not because innovation is bad, but because it's really difficult to find anything in the kata that has not been boiled down to the essence of effectiveness.
Most koryu (again, pre-1868 traditions) kata are paired kata, always practiced with a partner. The reasons for doing the kata a particular way become vividly clear in a bright black and blue manner if you try to change things. The attacking partner is an immediate check to see if what you are doing is effective or not. And when it's not, you may well end up with a beautiful bruise as proof. Recently a friend and I spent a morning working through some kata slowly. Each time we tried to change the kata, we discovered that the kata form was the strongest way of responding for both the shitachi and the uchitachi. Each time we tried something different the openings and weaknesses of the new positions were clear. After hundreds of year of practice and examination, our forebears in the system had worked out the most effective way for things to be done. Our lesson was to understand why they designed the kata as they did.
The practice of the kata define the koryu traditions. Nearly all of the lore and wisdom that generations of teachers have accumulated is in embedded in the kata. It's up to students to tease this knowledge out. One way to do that is with what my friend and I were doing. You deconstruct the kata, try different reactions and attacks at each juncture and see if they work, or as we discovered, why they don't work.
Traditional Japanese systems, koryu budo, generally have very specific and clear pedagogy. Shinto Muso Ryu has a clear set of 40+ jo kata, as well as 12 sword kata, 12 walking stick kata, 24 kusarigama kata, 30 jutte kata, and I've forgotten how many hojo kata. These are very clearly defined. It's extremely difficult for teacher who hasn't been training for decades to make changes, and the kata themselves make it difficult. As I discussed above, we couldn't find any weaknesses in the kata we were exploring. We just learned a lot of options that don't work as well those taught in the system already. With this kind of situation, there just aren't many opportunities for innovation.
The most common way koryu arts change is that someone develops a new kata to address some situation or condition that is not considered by the existing kata. In Shinto Muso Ryu for example, they developed some new kata at the end of the 19th century to make use of the walking sticks that had become popular at the time. This is a logical extension of the principles of the stick that is the main weapon in Shinto Muso Ryu to a shorter stick. They didn't change old kata, or get rid of anything. They developed a few new kata to teach an understanding of the ranges and uses of the shorter stick. Systems do change, but they do so very slowly. With koryu, those changes are usually minor additions to the system rather than revolutions in the way things are done.
People sometimes wonder why koryu systems don't have lots of sparring and tournaments like the modern arts of kendo, karatedo and judo. Surprisingly, this is not a new question. Groups have been arguing about the value of sparring type practice in Japan for over four hundred years. When Japan was at war with itself, which was most of the time from about 1300 through 1600, there were more than enough opportunities for people to test their ideas, techniques and skills, so the question didn’t come up. Once Tokugawa Ieyasu unified the country and removed the last possible source of revolution in 1615, those opportunities disappeared. Soon after that sparring and challenge matches started to appear. Arguments over the value of sparring compared with kata training began almost immediately, and have continued unabated to this day. Over the centuries though, the styles that emphasize sparring as a part of their training never demonstrated significantly better records in the many challenge matches. If the sparring faction had shown consistent success the other systems would have changed rather than lose. The systems that emphasized kata weren’t losing, so there was no need to change. Kata remained the core of training because when done properly, it works.
Tournaments are a relatively recent phenomenon. Tournaments first showed up late in the 19th century once the Japan had reformed its government and sword teachers had no way to make a living. Some people started doing matches to entertain the public and try to support themselves as professional martial artists after traditional positions working for daimyo disappeared.. These didn't last long, but they contributed to the development of modern kendo. Modern kendo equipment dates back to that used for sparring and some challenges as early as the 17th century.
Sword demonstrations and prize matches during the Meiji Era (1868-1912) popularized and contributed to the creation of a sport form of kenjutsu done with shinai (bamboo swords). Similar matches for jujutsu schools contributed to the rise of Kodokan Judo. Kano's students won a number of noted victories and the Kodokan was invited to participate in inter-style matches by the Tokyo Police. The Kodokan did exceptionally well in most of these matches and earned an impressive reputation. These matches though also drove some significant changes in the Kodokan's curriculum.
Fusen Ryu is reported to have defeated a number of Judo representatives with strong ground techniques. At the time, Kano was not in favor of focusing on ground fighting because he felt it was a dangerous place to be in a street fight. However, these losses on the ground in public matches pushed him to develop a groundwork curriculum for Judo. One of the big surprises about this is the way he went about it. Contrary to the idea of martial schools jealously guarding their secrets, at this time at the end of the 19th century, people were much more open. Kano invited the head of Fusen Ryu to teach groundwork at the Kodokan Dojo, and he did. With the help of the head of a rival system, Kano significantly strengthened the Kodokan curriculum. Kano never became a huge fan of groundwork, always believing that staying on your feet was optimal in a fight, but the pressure of doing well in competitive matches drove him to adapt his art.
In addition, Kano changed from the classic menkyo, or licensing, system, and created the modern dan rank system based on competitive ability. The koryu systems award licenses based on a persons level of understanding and mastery of the system, up to and including full mastery of the system. Kano abandoned this system for one in which students were ranked according to competitive ability in matches. If a student could defeat four other students of 1st dan level (commonly known as black belt) , then he was promoted to 2nd dan (black belt). This resulted in tremendous changes in what is taught and how students train. Anything that is not allowed in competitive matches is marginalized in training, even if it is effective in combative situations outside of training. The focus narrowed to those techniques which are most effective in competition. The up side of this focus is that it drives innovation and experimentation. Judoka are constantly looking for innovative ways to win in competition and refining their techniques to make them more effective. The down side is, as I describe above, that anything not useful in competition is largely ignored, even if it is highly effective in situations outside of competition.
Various pressures on competitive martial systems are still visible today. For the larger systems such as Judo and various Karate styles, two of the big pressures are popularity and money. In the last 15 years the International Judo Federation has been busy making numerous changes to the rules for competitive Judo matches in order to make Judo more television friendly to maintain popularity and keep it's place in the Olympics. The matches are seen as being too slow and difficult to follow, so changes were made to speed things up. In addition, there seems to be some reservations about how well people from other systems, such as wrestling and BJJ, do when they enter Judo tournaments. I have heard complaints that wrestlers and BJJ players use a lot of leg grabs and take downs that aren't classical Judo. The techniques work though. My feeling is that in Judo, we are reacting in the worst way possible to these challenges from wrestlers and BJJ players. Instead of inviting them into our dojo to learn from them, as Kano did, the IJF has chosen to ban the leg grabs and take downs from Judo competition. To me this only makes Judo weaker and less worthy of study.
In the Karate world, I see a lot of things in tournaments where combative functionality is not even considered. People invent kata that are flashy and athletic, but have nothing to do with the rich history and combative effectiveness of the Okinawan traditions. I have seen rules for weapons kata that require a certain number of weapons releases. This means that people are required to throw their weapon into the air! From a standpoint of combative functionality, this is ridiculous. However, to people who don't know better, this looks impressive. These Karate tournaments seem to be responding to a desire to be as popular as possible, rather than as effective as possible. It is a similar to what the IJF is doing make Judo more television friendly so the International Olympic Committee won't drop Judo from the Olympics like it tried to do with Wrestling a few years back. I won't even get into the silliness that is Olympic Tae Kwon Do.
Many of the modern arts are relatively easy to change because they are competition focused and committee governed, so changes in the rules will drive major changes in training. The koryu arts are deeply seated in kata that have been refined over centuries, and I can't really imagine any pressure big enough for them to make significant changes to their curriculums. Since the classical systems are not looking for rapid growth or tv money, they are under no pressure to change except that which they have always had; to adjust their systems to they remain relevant to the world around them. Judo and Karate both have strong depths of kata, well thought out and highly refined, but these traditional, effective and functional kata are often ignored in the race to perform well in competitions. The desire to do well in competition and to be visible on the world stage will continue to drive changes in these arts. I would love to see the pressure and focus of modern arts return to combative functionality, but I doubt that will happen when it is so easy to get caught up in the ego trap of popularity.