[x,v,s,h,f]=cont(@homoclinic, x0, v0, opt)The homoclinic curve file has to know
In the GUI-version MATCONT, you call do this by selecting the cycle with large period as initial point, and then defining it (Type Initial Point) as a Homoclinic-to-Saddle.
For homoclinic-to-saddle-node orbits, the user needs to replace Hom with HSN in the above commands. Note that there are no HSN orbits emanating from a BT point, so only the initializers from a limit cycle and a previously computed HSN are available.