'Magic' was the Allied (primarily US) cover designation for the decryption and analysis of Japanese diplomatic signals (1941/45).

During World War II the codename was also used for decrypted Japanese military signals and, adding to an inevitable level of confusion, all these deciphered messages were classified as 'Top Secret Ultra'. While there was considerable overlap in substance as well as designation between the diplomatic and military categories, it is important to preserve the distinction between messages exchanged by the Japanese foreign ministry and its diplomatic posts abroad and those of the Japanese army and navy, and accordingly 'Magic' is here limited to its more generally understood meaning of diplomatic communications, and Japanese military decipherments are covered in 'Ultra'.

In its most basic terms, 'Magic' was an Allied cryptanalysis project which involved the US Army’s Signals Intelligence Section (SIS) and the US Navy’s Communication Special Unit. 'Magic' was established to combine the US government’s cryptologic capabilities in one organisation dubbed the Research Bureau, in which intelligence officers from the US Army and Navy (and later civilian experts and technicians) were all located under one roof. Although they worked on a series of codes and ciphers, their most important successes involved 'Red', 'Blue' and 'Purple'.

In 1923, a US Navy officer acquired a stolen copy of the secret codebook used by the Japanese navy during World War I. Photographs of the codebook were given to the cryptanalysts at the Research Desk and the processed code was kept in red-coloured folders (to indicate its Top Secret classification), and as a result the code was designated 'Red'.

In 1930, the Japanese created a more complex code that was codenamed 'Blue', although 'Red' was still being used for low-level communications. It was quickly broken by the Research Desk no later than 1932. US military intelligence listening stations began monitoring command-to-fleet, ship-to-ship, and land-based communications.

'Magic' included all decrypted messages in Japanese diplomatic codes and ciphers. The most valuable by far were those encrypted by the cipher machine known to the Americans as 'Purple'. The cryptanalytical feat of breaking into the 'Purple' ciphers was extraordinary.

Japan was an ally of Germany under the terms of the Tripartite Pact signed on 27 September 1939, and in the autumn of 1939 Germany government began to supply Japan with technical assistance in the task of upgrading its Far Eastern ally’s communications and cryptography capabilities. One element of this effort was the despatch of modified Enigma machines to secure Japan’s high-level communications with Germany, and the resulting traffic in the new 'Purple' code baffled US cryptographers.

Like Enigma, 'Purple' began its communications with the same line of code but then became an apparently impenetrable jumble. Codebreakers tried to break 'Purple' messages by hand, but found that they could not do so. The codebreakers then realised that the new code was not a manual additive or substitution code like 'Red' and 'Blue', but a machine-generated code similar to the German Enigma cipher. Decoding was slow and much of the traffic was still hard to break, and as a result by the time the traffic was decoded and translated, the contents were often out of date.

A reverse-engineered machine created in 1939 by a team of technicians led by William Friedman and Frank Rowlett could achieve some success with decrypting 'Purple' messages by replicating some of the settings of the Japanese Enigma machines. This accelerated the decryption process, and the addition of more translators in 1942 made it bother faster and easier to decrypt intercepted messages.

The 'Purple' code was first used by Japan in 1940. US and British cryptographers had broken some 'Purple' traffic well before the 'Ai' attack on Pearl Harbor. However, the 'Purple' machines were used only by the Japanese foreign ministry to carry diplomatic traffic between Tokyo and Japanese embassies. The Japanese Navy used a completely different system known as JN-25.

US analysts discovered no hint in the 'Purple' traffic of the impending Japanese attack on Pearl Harbor, which is hardly surprising as the Japanese had exercised great care not to discuss their plan in foreign ministry communications. In fact, no detailed information about the planned attack was even available to the foreign ministry as this was seen by the military, and most especially its more nationalist members, as insufficiently reliable. US access to private Japanese diplomatic communications (even the most secret ones) was less useful than it might otherwise have been because policy in pre-war Japan was controlled largely by military groups and not by the foreign ministry. Moreover, the foreign ministry itself deliberately withheld from its embassies and consulates much of the information it did possess, so the ability to read 'Purple' messages was less than definitive regarding Japanese tactical or strategic military intentions.

Throughout the war, the Allies routinely read both German and Japanese encrypted signals. The Japanese ambassador in Germany, General Baron Hiroshi Oshima, often sent invaluable German military information to Tokyo, and this information was routinely intercepted, decrypted and therefore available to be read by President Franklin D. Roosevelt, Prime Minister Winston Churchill and a small but very select group of the Allies' most senior military commanders. The Japanese considered the 'Purple' code to be totally unbreakable, and most went to their graves refusing to believe that it had been broken by analytic means and that a person must therefore have betrayed the system.

In the period before 7 December 1941, the decrypted diplomatic information was of limited value to the USA because of the manner in which it was disseminated within the US government. 'Magic' was distributed in such a way that many policy makers who could have made use of the information knew nothing of it, and those to whom it actually was distributed saw each message only briefly as the courier stood by to take it back, and in isolation from other messages as no copies or notes were permitted. Before Pearl Harbor, recipients saw only those decrypts thought 'important enough' by the distributing army or navy officers. Nonetheless, their ability to read 'Purple' traffic gave the Allies a great advantage: for instance, Oshima produced long reports which included reports on personal discussions with Adolf Hitler and a report on a tour of the ant-invasion defences in northern France. General George C. Marshall, the US Army chief-of-staff, later said that Oshima was the Allies' main basis of information regarding Hitler’s intentions in Europe.

Once they had the capability, the Americans were able to read the most secret Japanese diplomatic communications from a time before Pearl Harbor in December 1941 to the end of World War II. 'Magic' supplied no specific warning of the attack on Pearl Harbor or on the American and British possessions in South-East Asia, but the cumulative effect of the 1941 messages was the impression of an expansionist Japan ever nearing a decision for war. With 'Magic' US officials could in effect peer over the shoulder of the Japanese ambassador in Washington as he sought a diplomatic formula to avoid war in the spring and autumn of 1941. By way of the Japanese ambassador in Berlin, 'Magic' intelligence also provided vital information throughout the war about German plans and operations against the USSR and the Western Allies, as well as about Japan’s relations with the USSR and its attempts to secure Soviet mediation in ending the war.

The 'Purple' cipher machine consisted of two typewriter keyboards connected by a maze of circuits, plugs and switches. Machine encipherment was an important form of communications security in World War II, the most famous example being the German Enigma machine, but it was unusual for Japan and the 'Purple' itself was unique. No 'J' machine, called the Type 97 Alphabetical Typewriter, survived the war, nor are any of the US analogues known to exist. Eyewitness descriptions are few and sparse but offer a rough idea of how the machine worked. In place of rotors, which supplied a sequence of letter substitutions in most enciphering machines, the 'Purple' machine used switching gear (stepping switches) from the dial telephones of the day. To encipher, the operator pressed the appropriate typewriter key for the plain text letter. This carried current through a plugboard, which provided changeable letter substitutions, which served as 'keys' for use on any given day. Thence the current ran through a series of stepping switches. Each of these consisted of a semicircular array, or matrix, of electrical contacts facing a shaft from which projected finger-like conductors, which rotated across the matrix. Each contact and finger stood for a letter of the alphabet. Since the fingers and the matrix were wired differently, each time a typewriter key was pressed a letter substitution occurred. At set intervals the shaft and fingers rotated ahead one or more steps to a new set of contacts, new wiring, and deeper encryption. The current then passed on to another stepping switch (the 'Purple' machine had four) and finally depressed a cipher text key in the second typewriter.

These successive substitutions provided a formidable challenge to cryptanalysts. Theoretically, the possible substitutions by machine cipher were almost endless, but practically the task was somewhat less daunting. In most cases, including 'Magic', the cryptanalysts found beach-heads into the cipher from bureaucratic words and phrases regularly used and available in plain text. Certain forms of address and key words related to the events of the day could be anticipated.

In the case of 'Magic', the Japanese foreign ministry made a critical mistake in repeating messages sent in previous encipherment (by the so-called 'Red' code), so old decrypts could be used to solve 'Purple'. Easing the task was the division of the alphabet into two subsets, each group enciphering separately. Keys changed every 10 days but within the month varied only slightly and predictably. In many cases in World War II, cryptanalysts were assisted by the capture of the other side’s code or cipher material or machines, but this was not the case with 'Purple'. However, US cryptanalysts could get the gist of some enciphered texts from Japanese diplomatic messages delivered to the state department.

Leading the attack on 'Purple' was the best US mind in codes and ciphers, William F. Friedman, chief cryptanalyst of the US Army Signal Intelligence Service. Though not trained specifically as a mathematician, Friedman was expert in statistics and probability, and an authority in applying these to cryptanalysis. In addition he brought to the task of breaking 'Purple' mastery of the whole field of cryptology, exceptional intuition and dogged perseverance. Although his contribution was vital (and so intensive it put him in hospital with a nervous breakdown), this was not a one-man show, for Friedman engaged talented PhDs in mathematics to help him.

The US Army and US Navy had made a prolonged effort to gather and train staffs of cryptanalysts, pooling resources and sharing results in the attack on 'Purple'. Beginning in early 1939, the breaking of 'Purple' took 18 months. A critical breakthrough occurred when a cryptanalyst from naval intelligence, Harry L. Clark, suggested that the Japanese might be using ordinary telephone stepping switches instead of rotors. Laboriously they separated cipher text into segments representing different key settings and then tackled texts in the same key, building out from known letters and words, looking for symmetries in the position of letters, and trying out letters according to the known frequency of their use. Translators filled in missing letters and completed words. With agonising slowness at first and then gradually more swiftly the plain text messages emerged.

The first message was completed on 25 September 1940, two days before the signature of the Tripartite Pact. So impressed with the feat was one authority that he referred to the team as 'Magicians', which led to the overall codeword. Once the Friedman group understood what kind of a machine enciphered 'Purple' and how it must be wired, they constructed a machine to duplicate its functions. By the spring of 1941 four of these machines were at work, one in the Philippines, two in Washington, and one in the UK at Bletchley Park.

This US gift of a 'Purple' machine to the UK, revealing to a foreign power a vital state secret, was the first major step in establishing British and US co-operation and co-ordination in signals intelligence and cryptanalysis.

Upon receipt of the 'Purple' machine the British began intercepting and decrypting the Japanese foreign ministry’s messages to and from its embassies and consulates in Europe and the Middle East. By June 1941 the British had received a second machine for use in Singapore. One example to which these machines were put to use was to reveal the treachery of Burma’s prime minister, U Saw. On his way back home from London, after unsuccessful talks about Burma’s independence, he visited the Japanese consulate in Lisbon. He assured the consul general that if the Japanese invaded Burma his people would rise against the British and help the Japanese drive them out. On the following day the Japanese foreign ministry was informed of this conversation; the encoded signal was decrypted; and U Saw was arrested further along his journey and spent the rest of the war in internment.

The flow of decrypts continued until Japan’s defeat, and the Japanese never suspected that their most secret diplomatic cipher had been compromised. Some messages were decrypted and translated the same day and most within a week; a few in cases of key change took longer, one as long as 59 days. Shortage of translators, in particular those familiar with the forms used in official, telegraphic transmissions, caused delay.

'Magic' was, as the US Army’s chief-of-staff, General George C. Marshall, said, a 'priceless asset' for the USA and UK, and extraordinary measures were taken to keep it secret. Indeed, these precautions were so protective, at least before Pearl Harbor, that they hampered effective use of the information. Horrified to find a copy of a 'Magic' message in a White House waste bin, the US Army for a time struck the president off its list of recipients.

'Magic' was treated with such secrecy that it was almost impossible to integrate it with other forms of intelligence. In fact before Pearl Harbor there was no national system for correlating and evaluating intelligence from different sources. By the end of the war the distribution system was systematic and comprehensive, the president and high officials receiving the daily 'Black Book', a digest of important 'Magic' and 'Ultra' intelligence from British and US sources. Although it revealed the imminence of war, 'Magic' did not pinpoint Pearl Harbor or other objectives since Japanese diplomats were kept in the dark about military plans. However, a better organised US intelligence system might have been alerted by a message of 24 September 1941, not in 'Purple' but a lesser cipher, asking the precise location of warships in Pearl Harbor. But distinguishing Japanese intelligence-gathering for an attack on Pearl Harbor from the mass of information sought by the Japanese on American naval activities throughout the Pacific would have been difficult at best. In the last hours before war, 'Magic' did disclose the Japanese intention of breaking off negotiations in Washington and the particular hour this was to occur, 13.00 in Washington and dawn in the Hawaiian islands. Washington officials anticipated an attack somewhere and issued warnings, but missed the Hawaiian connection.

While 'Magic' had limited operational value during the war, it was important in reinforcing US and British perceptions of Japanese aggression. Intercepts of June and July 1941 gave an inside view of Japan’s coercive diplomacy to secure military bases in southern French Indo-China. They also plainly indicated Japan’s interest in further penetration of South-East Asia: the foreign ministry directed its consuls to find Japanese who knew Malaya, secure maps of the region, and gain information about the beach defences of the Netherlands East Indies and the camouflage and markings of US warplanes in the Philippine islands group.

Typical as a preliminary to war were messages ordering Japanese consulates to destroy back files and cipher material. They indicated, too, plans for anti-US propaganda and espionage networks in Latin America and the recruitment of African Americans as spies.

'Magic' was also an excellent source of intelligence on the European war. While the Japanese foreign ministry had limited access to information about its own military forces, and in any case shared it only sparingly with its missions abroad, traffic in the opposite direction was considerable: Japanese embassies and legations in German-occupied Europe, Berne, Lisbon, Stockholm and Moscow provided a stream of information on the tide of battle and German capabilities and intentions. The military and naval attachés in these posts used their own codes but even these in time were decrypted.

As noted above, the most valuable reports were those of Oshima, the Japanese ambassador in Berlin. As principal representative of Germany’s Axis partner, Oshima had access to the highest German sources including Hitler himself, as well as to leaders of the German armed forces. The Germans were not overly generous in sharing secrets with their Japanese ally, but neither could they leave Tokyo entirely in the dark, so the ambassador’s reports were of vital interest to Washington and London. Early in June 1941 Oshima reported that conversations with Hitler and Foreign Minister Joachim Ribbentrop indicated, in all probability, the imminence of a German attack on the USSR. Although the massing of German forces in the east was impossible to conceal or ignore, British and US intelligence found it difficult to believe that Hitler would actually strike. Rather it was suspected that he was seeking to intimidate Stalin into making large territorial concessions. Oshima’s message was important, though by no means singular, in convincing the doubters of Hitler’s real intention.

The course of the swaying tide of battle in the German/Soviet war was of intense interest to Japan, and Oshima followed it closely. By way of his decrypted messages the western powers gained confirmation from Berlin that the German drive on Moscow in the autumn of 1941 was slowing, and the following spring that the Soviet counter-offensive was ebbing. During the crucial battle for Stalingrad, word that Japan had rejected a German appeal to attack the USSR encouraged the view in London that the Soviets would hold out. Further examples are a decrypt of August 1943 reflecting German pessimism during the great battle of Kursk, and another in January 1944, as the Allied 'Overlord' invasion of France approached, dwelling on the difficulty Hitler saw in waging war on more than one front.

One of the most valuable contributions of 'Magic' was the information which Japan’s embassies in Germany and Vichy France provided about German defences and troop dispositions against the Allied invasion of France in June 1944. Reports by Oshima and his naval attaché about their tours of the defences in France gave details of the German command structure in the west, the number of divisions in each sector, the composition of the mobile reserve, the nature of the Atlantic Wall, and the presence of underwater obstacles erected against landing craft. The Germans planned to defend at the beach line, said Oshima, and smash any beach-head with their Panzer reserve. As to the point at which the Allied forces would land, and whether or not the landings in Normandy were to be the only landings, the Japanese confirmed German uncertainty right down to D-Day and beyond.

Also of great value were decrypts concerning German production, morale, and weaponry. By way of the Japanese, the Allies learned the characteristics of the new Schnorchel-fitted U-boats, and specifications of their radio-guided, air-launched and rocket-propelled anti-ship weapons.

The growing weight of the Allied strategic air offensive against Germany was also reflected in 'Magic' decrypts. In August 1943 Oshima told of German plans for increased fighter aircraft production to counter the raids. By June 1944 the embassy was describing daylight attacks as overwhelming. It reported more than once on severe and possibly fatal bomb damage to oil refineries and synthetic oil plants. The Berlin embassy also correctly predicted, more than once, the German 'Wacht am Rhein' offensive in the west late in 1944, but not where, precisely when, and in what strength it was to unfold.

'Magic' was only one of many intelligence sources available to the Western Allies. It was not always respected or heeded: to some, the Japanese seemed gullible, taken in by German claims. Nevertheless, it provided an extraordinarily valuable supply of operational intelligence both in Europe and in waging the Pacific war. Its final gift was the revelation to the Americans of Japan’s futile effort to secure Soviet mediation in ending the Pacific war, which clearly indicated Japan’s insistence on maintaining the retention of the Emperor Hirohito as a condition for peace.