In AEM, you can swap out yui engine with GCC (Google Closure Compiler https://github.com/google/closure-compiler-js).
For example, see these client library nodes that use GCC for js:
If you want to implement your own custom script processor, then you would implement this java interface:
This allows you to define ScriptProcessors that can modify script sources:
- the processors are configurable with options
- the processor can be used for minification, but also for non-minified cases
- The default ScriptProcessors options can be set via the HTML Library Manager OSGi configuration (in the /system/console/configMgr UI).