Package shlex implements a simple lexer which splits input in to tokens using shell-style rules for quoting and commenting. The basic use case uses the default ASCII lexer to split a string into sub-strings: To process a stream of strings: To access the raw token stream (which includes tokens for comments):