I wrote some lecture notes that include the line intersection formula, as well as a bunch of other geometric constructors. You can download it from

http://www.cs.berkeley.edu/~jrs/meshpapers/robnotes.ps.gz

(in gzipped PostScript form; see Section 3.6). I believe this is the simplest possible formula. The notes also contain some tips on getting a numerically stable answer despite floating-point roundoff.

The notes don’t address how to tell whether the segments intersect, so I’ll address that here (and I should add it to the notes next week). If your segments are ab and cd, and they’re not parallel (see the notes for how to tell that), then they intersect if and only if

Orient2D(c,d,a) x Orient2D(c,d,b) <= 0 AND
Orient2D(a,b,c) x Orient2D(a,b,d) <= 0.

The function Orient2D() is described in the notes, and I have C code for computing it exactly from floating-point input at

http://www.cs.cmu.edu/~quake/robust.html

If the lines are identical, then you determine the segment intersection by comparing x-coordinates (y-coordinates if the lines are vertical).

- If the calculation for the line between one of the pairs of endpoints comes up (0,0,0), the two points are equal. In this case, there is only an intersection if one of these points lies on the line through the other two points, which can be tested by taking the dot product of the point’s coordinates with the line’s.

- If the calculation for the point where the two lines meet comes up (0,0,0), the two lines are coincident. In this case you can test for an intersection by comparing the order of the points’ coordinates.

- As I said in my original comment, if you get a divide by zero converting back to Cartesian coordinates, the lines are parallel. In this case there can be no intersection.

- Finally, once you have the intersection point in Cartesian coordinates, you can test the ordering of its coordinates against the original points to determine whether it lies between them.

But, if you want to perform exact comparisons such as this (do you really have (0,0,0) or just a vector very close to it?) you probably need to be using multiprecision integers, and avoiding floating point, or your results will not be reliable. Or, to put it another way: if you think approximate calculation with floating point is good enough, why are you worrying about cases like two coincident lines that only make sense for exactly defined numbers?

For another. Why not go parametric? You represent your lines as a pair of functions R->R. To make it work with being segments, let them be represented as functions [0,1]->R .Thus, if a line goes from (x1,y1) to (x2,y2), then this is represented by the pair
x: t -> x1+t*(x2-x1)
y: t -> y1+t*(y2-y1)
Now, the vertical line has t -> x1, a constant function. The point has both functions constant. And finding the intersection of the lines described by (x1(t),y1(t)), (x2(t),y2(t)), respectively, is the matter of finding t1 and t2 such that x1(t1)=x2(t2) and y1(t1)=y2(t2). This yields a system of linear equations, easily solved, and with a final check for whether the solution stays within the constraints given.

The method is not hard to adapt to rays and lines - just adjust permissible solutions to be positive or all of them, respectively.

I’m going to try for an algorithm without many case splits. This is inspired by your description of the algorithm in Sedgewick, but it does give you the intersection point.

I’ll use some special notation: “.” is the dot-product operator; if A is a point then Ax is the x coordinate; and if A,B,C are points then Area(ABC) is the signed parallelogram area: (Ax-Bx)*(Cy-By) - (Ay-By)*(Cx-Bx) (then |Area(ABC)| is the area of a parallelogram where 3 of the vertices are A,B,C; and Area(ABC) = -Area(CBA)).

We’re testing whether the line segments AB and CD intersect.

First, if A=B and C=D, then we actually have two points; check whether A=C. (first case split)

Now, assume without loss of generality that A!=B. (second case split)

Next, compute Area(CAB) and Area(BAD). If the product of these numbers is negative, then C and D are on the same side of AB; report no intersection, and exit. (third case split)

If Area(CAB) and Area(BAD) are both 0, then A,B,C,D are all collinear; jump to the CollinearIntersection(AB,CD) routine below. (fourth case split)

Now we know that the lines AB and CD are neither identical nor parallel, so we can find a unique intersection point E. (Note that I said “lines”, not “line segments”; we still don’t know if the line segments intersect.) This intersection point is:

E = C + (D-C)*Area(CAB)/(Area(CAB) + Area(BAD))

(Due to the above case splits, we know that we are not dividing by 0 here.)

Now, A,B,E are collinear; jump to the CollinearIntersection(AB,EE) routine immediately below.

Now we are testing CollinearIntersection(AB,EF) (where EF is either CD or EE, depending on how we got here); we know that A!=B, and that A,B,E,F are all collinear.

Define UNIT=(B-A).(B-A).

Compute dE = (E-A).(B-A)/UNIT, and dF = (F-A).(B-A)/UNIT.

If dE and dF are both less than 0, or both greater than 1, then report no intersection, and exit. (fifth case split)

Now, sort the 4 numbers 0,1,dE,dF; remove the least and the greatest number; and call the remaining middle two numbers dG and dH. (several more case splits, depending on how you count)

Compute G = A + dG*(B-A) and H = A + dH*(B-A). The intersection is the line segment GH (which may be a single point).

Finished! After 5 case splits (into 6 cases), plus a 4-element sort (which you can decide to count as several case splits, or not, as you please).

Of course, in the real world, there’s more to worry about geometric algorithms than just this. Are you computing over the rationals (in which case this algorithm should work perfectly), or over floating-point numbers? If you are computing over floating-point numbers, and you pass in coordinates which are collinear, is it acceptable to sometimes say that there is no intersection, or that there is a single-point intersection, rather than that there is a line-segment intersection? (If you want an exact answer using floating-point arithmetic, then you’ve got a lot more work ahead of you!)

I guess what I’m suggesting is that maybe first you can figure out whether one line segment crosses the line made by extending the other, as an easier problem?

My other thought would be to project the one line onto the other, and see if the projected… Uh, I’m not sure where I’m going with that. Wait, if the projection doesn’t overlap (which is easy to check), then the line segments don’t cross?

I don’t really know if that gets you any closer, but those would be the first two things I would try.

http://planetmath.org/encyclopedia/HoughTransform.html

In normal form, a line is defined as:
x cos(theta) + y sin(theta) = rho.

That will eliminate the problem of infinite slopes.

(I think that would work. Didn’t test it though.)

Cartesian coordinates of points to projective points:
(x,y) => (x,y,1)

The projective line ax+by+cz=0 through two projective points:
(y0z1-y1z0, z0×1-z1×0, x0y1-x1y0)

The projective point where two lines intersect:
same equation as the previous one, with the variable names changed

Projective back to Cartesian:
(x,y,z) => (x/z,y/z)

If you get a divide by zero in the last step, the lines are parallel.