This extension is EXPERIMENTAL. The behaviour of this extension -- including the names of its functions and anything else documented about this extension -- may change without notice in a future release of PHP. Use this extension at your own risk.
See also the appendix about tokens.
Beginning with PHP 4.3.0 these functions are enabled by default. For older versions you have to configure and compile PHP with --enable-tokenizer. You can disable tokenizer support with --disable-tokenizer.
The windows version of PHP has built in support for this extension. You do not need to load any additional extension in order to use these functions.
Note: Builtin support for tokenizer is available with PHP 4.3.0.
The behaviour of these functions is affected by settings in php.ini.
Table 1. Tokenizer Configuration Options
The constants below are defined by this extension, and will only be available when the extension has either been compiled into PHP or dynamically loaded at runtime.