How to restrict the file size of uploads in Touch UI and Classic UI
How do I modify the upload file size restrictions in AEM Assets UI to prevent users from uploading larger files than a certain number of bytes?
See this documentation regarding setting sizeLimit property to restrict max file size for asset upload.
Unfortunately, in Classic UI there is no such feature.
This blog post mentions how you can customize the product to do this:
However, note that this customization is not supported by Adobe Customer Care. So if you implement this customization, then you would have to maintain it as part of your code base.