Thanks for your question. The first caveat is that this varies by league rules. In travel, high school, college, etc. it is on pitcher's release. That is why you see runners leaning back and mimicking the windmill and throwing themselves forward as the pitcher's arm goes forward to leave the bag at the same time the pitcher releases. This helps them time the release as well as get some momentum when coming off the bag. See this page on base running.
Half way down you will see a picture of the baserunner with her arms behind her with the heading: "Getting and Timing a Lead Off the Base".
That should clarify what I am talking about.
I said "most" initially on pitcher's release, because some younger leagues may allow stealing only after the pitch crosses the plate. Others may permit stealing only a certain number of times per inning. This is to help the catchers develop and give them a fighting chance.
So you know, I have never seen stealing permitted during the wind-up. Only at the release or later. If the runner leaves the base before the release, the umpire is suppose to call the runner out.