Thanks you very much Patrick - the technical articles were very useful and allowed my to investigate and test many things. I have done a lot more testing and investigation and I cannot, unfortunately, find any identifiable problems.
Symptoms are as follows:
1) When starting from dead cold (overnight), if the engine doesn't start on the first crank, then it is very hard to start. While cranking, there are frequent 'backfires' that cause the engine to stop dead. It is as though the ignition fires at the wrong time in the cycle and causes the spinning engine to freeze at that point.
2) Once started from dead cold (which often happens on the first crank, despite the above), the engine idles very smoothly for the first 60 seconds or so. After that, the engine starts to miss while idling, which gets worse as the engine warms up.
3) Once at normal operating temperature, the engine misses badly while idling. It idles smoothly for a second or two, then misses, which causes the revs to drop suddenly, and then the control system seems to increase the idle speed to counteract the drop in revs for a second, and then the cyle repeats. Turning on the air conditioning makes this cycle much worse. That is, the miss is worse with the airconditioning engaged, and will frequently stall the engine. Listening at the exhaust pipe, you can hear a 'phut, phut, phut' noise when the ening is missing.
4) After idling at intersections and the like, there is a very bad hesitation or stumble when accelerating away. The engine will nearly stall for 2 to 4 seconds, and then take off. Occasionally it does stall. This bad hesitation also happens when driving along at normal road speeds if the foot is taken off the accelerator for a few moments (throttle fully closed) and then accelerate again.
5) While driving at normal road speeds (after the hesitation) or accelerating and the throttle is not fully closed, there is no miss whatsoever, and the engine performs very smoothly. While ever the throttle is partially or full open, the engine performs very smoothly and with normal power, apart from the initial hesitation.
6) While stationary with the airconditioning off, if the accelerator is depressed a tiny amount to increase the revs to about 1000 rpm (ie 200-300 rpm above normal idle) the engine will run very smoothly with no miss. This may be because the throttle position sensor is signalling just off idle.
7) If the throttle position sensor is disconnected the engine will idle very smoothly with no miss. It will also run at slightly higher revs, but if the idel speed is adjusted down to anout 700rpm again it still runs perfectly smoothly with no miss. I discovered that one of the effects of disconnecting the TPS is that the spark advance is increased significantly by the ECU. It looks as though it goes from the normal 12 degrees up to about 20 to 25 degrees BTDC. I don't kow what other compensating action is taken by the ECU when it detects a fault with the TPS after it is disconneted.
Replaced plugs, spark plug leads, distributor cap and rotor, fitted brand new distributor. Checked for any vacuum leaks. Removed throttle body and thoroughly cleaned and checked it. Checked many other sensors for correct operation and voltages (where possible) at the ECU. This include the water temperature sensor, air flow sensor, air temperature sensor, throttle postion sensor.
Checked that the spark plugs are connected to the distibutor in the right order, checked that the TDC mark on the crank is actually top dead centre (measued piston position through the spark plug hole), and TDC is aliged in the distributor with spark plug for cylinder number 1.
Checked that the exhaust gas recirculation valve is operating by disconnecting the normal vacuum hose and applying vacuum while idling. This caused the engine to idle worse, although only a bit worse than it already was. The change was noticeable, however.
Fitted a different ECU, with no change to symptoms at all. Fuel pressure was checked by the mechanic.
I was worried that the problem may be caused by low or uneven cylinder compression. However because the engine will idle perfectly smoothly in some cases (e.g. TPS disconnected) this would not be the case if the problem was low compression. Similarly if the problem was broken leads or damaged spark plugs, again doing simple things like very slightly opening the throttle would not cause the symptoms to disappear. Also, because the engine runs so well (smooth and with plently of power) right up through the rev range, even under heavey load, it would seem unlikely that the problem is simple spark plug miss-firing.
Some ideas about what could possibly display such symptoms:
1) Igniter patially failing. I can not test the igniter, and I have not tried a replacment. While it obvioulsy still works to some degree, I wondered whether it is sometimes firing the coil at the wrong time. Because it is partially controlled by the ECU, sensing changes such as the TPS off idle could potentially affect the opertion of the igniter. Firing the coil at the wrong time would explain the bad backfire while cranking.
2) Cold start injector. If the cold start injector continued to operate instead of shutting off, then too much fuel would be injected while idling (and throughout the rev range), but the effects would probably be worse when there is very little air flow such as when at idle. This would also explain why while driving at normal speed, a period of time coasting with the trhottle closed could build up the fuel mixture in the manifled, which causes the hesitation when the throttle is opened again. It could also explain why if the engine does not start first time, it is very hard to start afterwards.
3) EGR staying partially open. This is supposed to shut off at idle, and in some oether conditions. If it was stayig partially or fully open at idle, the mixture would be wroing (too lean?) causing the bad idle.
Does anybody have any suggestions about these ideas or anthing else to try?
Today I removed the EGR, and checked that it was operating correctly. In particular I checked that it was normally fully closed and formed a good seal. All checked out perfectly. I gave it a thorough clean with carby cleaner anyway. So that rules out the EGR valve.
While there I also removed the cold start injector and checked that it was not spraying fuel all the time. It also was operating correctly, ruling it out as the problem.
Double checked the TPS wires and signals and it is operating perfectly as well.
So now I am back to square one, and have no idea what could be causing the problem. I went for a drive after reassembling everything, and the engine stalled at least 6 times - sometimes just because the idle was so bad, sometimes because the stumble on accelerating away from a standing start was so bad.
Pleas help if you have any ideas!