Tried this on our project, GitHub.com/hail-is/hail.
Sphinx is a persistent thorn in our side, but we've made it work. A couple things I had to do to get this working:
- something thinks a filename with dots (e.g. example.8bits.bgen) means there is a python module that needs to be imported (e.g. example), and fails when that does not exist; we have files like this in a data directory
- I could not quickly figure out how to ignore files, so I had to delete a Sphinx conf.py file.
- Sphinx resolves the _templates folder relative to where it is called, not relative to the doc string's source file, so I had to add several symlinks (rather than manually edit every doc string).
---
OK docs are generating. Quite slow, maybe a couple minutes. Looks like one core is at 100% during this process. We've got ~400 files. Maybe its parsing some data files? The whole directory is 318 MB.
---
Now I'm looking at the docs.
First thing I notice is that Python type hint forward references break everything. Consider:
class GroupedMatrixTable(
parent: 'MatrixTable',
...
The generated doc page is missing the class name and the entire class definition is inlined as a preformatted block (with highlighting :shrug:).
All my python function def's seem to be missing the function name?
No arg functions look really weird
def (
)
So all our docs are in ReST (thanks Sphinx :|). This means they're not valid Markdown. It seems that invalid markdown in a class doc string can break the formatting of all the methods (presenting them again as one reformatted block).
---
On the bright side, the mobile version looks great and the search works better than my experience with Sphinx.
Thanks so much for this great feedback! I haven't done any speed optimization on portray yet - as both at work and online I tend to organize my own projects as small self contained repositories. Clearly there is a lot of room for improvement there, I have a lot of ideas to make it faster.
And, yes write now Markdown only. I will say - this was a week only project: https://timothycrosley.com/project-2-portray so I think there's a good chance I could improve these points with one more week of time spent :)
Yeah of course! In retrospect my initial post seems a bit neutral to negative. I'm really glad this project exists, I'm endlessly frustrated by Sphinx. I'll keep my eye on your project!
A little off-topic, but as someone who's just started producing documentation in Sphinx, what are the problems/downsides that you've run in to?
I've used it for two projects so far: documenting a relatively small python module, and for documenting a RESTful API. I do a lot of embedded work, so I'd like to use it as a place to document a piece of hardware in it's entirety, from schematics/layouts to firmware and build toolchains to actual use of the device. Do you (or anyone else reading this) have any comments on that use case?
I think my main complaint at the moment is that it doesn't play nice with Markdown, so I had to re-format a bunch of pre-existing documentation for it to work in Sphinx. ReST seems to render alright in Gitlab though, so that's a plus.
My team at present use sphinx-doc[1] and generate both HTML and PDF. We write the documentation as part of the code using docstring and then other documents manually in rst. We use Sphinx plugin to automatically generate document from code.
I will give portray a try and see.
Is there any comparison between Sphinx-doc and portray?
We've historically done this as well. One issue we perpetually run into is that the docstrings get out of sync with the actual signature / types of the methods/etc that they purport to document. I suspect there are plugins that generate documentation from types such that there is less to keep synchronized. It is also unfortunate that Python projects have to set up infrastructure to build and publish documentation packages. I really like the effortlessness that Go's ecosystem brings--https://godoc.org just reads your source code from your git repository, so there is are no explicit steps for building and publishing documentation.
With Sphinx's Napoleon extension (https://www.sphinx-doc.org/en/master/usage/extensions/napole...) Python 3 type annotations will be added to the generated docstrings so you don't have to write the annotations both in the function definition and in its docstring. I've found this to be very helpful in keeping docs in sync with the code.
I use sphinx-apidoc[1] command to extract docstrings from code and create Sphinx documentation files out of them.
Enter this command to look for modules in a package, say, foopkg, and create .rst files to generate documentation for each module in the package and subpackages:
sphinx-apidoc --module-first -o docs/api foopkg
I am assuming that the Sphinx documentation project resides in a directory named docs. Each .rst file would contain automodule directives for Sphinx to automatically pull docstrings from foopkg and render them in the generated documentation. However, for Sphinx to understand these automodule directives, the autodoc[2] extension must be enabled in docs/conf.py.
extensions = ['sphinx.ext.autodoc']
If the documentation is written using Google style or NumPy style docstrings, then the napolean[3] extension must also be enabled.
The sphinx-apidoc command above would also write a file named docs/api/modules.rst which would be used to render the starting page of the automatically generated API documentation. So include this docs/api/modules.rst with the Sphinx toctree directive from the some other page that the user is likely to visit. For example, the start page of the project documentation would likely be rendered from `docs/index.rst`, so add this to `docs/index.rst`:
API
---
.. toctree::
api/modules
Now render the documentation with Sphinx normally:
This is great! You hit the need for configless docsites that many Pythonistas who are in data science and research have: we write a lot of experimental and prototype code. Docstring documentation in itself feels like a considerable investment in these contexts so anything that helps publication without any hassle is more than welcome.
Are you planning to support Restructured Text next to Markdown? I find ResT is used more often (in the machine learning ecosystem at least).
I just used this on a random project I happen to be building documentation for. I'm pretty in love with the minimalism of 'portray'. I wish there was a chance to get my team to use it, but I'll definitely be using it in my personal projects going forward!
portray requires your project to be packaged for it's auto discovery - if you get this error and your project has a setup.py or pyproject.yaml in the root (and this is where you ran portray from) let me know and I'll create a ticket! Otherwise - I highly encourage looking at using poetry to setup your pyproject.yaml file: https://poetry.eustace.io/
my project doesn't have a setup.py because it is not intended for packaging and it doesn't have a pyproject.yaml because to be honest, this is the first time I heard about such a file. But then again, the title says "zero-config".
Readit News
Sphinx is a persistent thorn in our side, but we've made it work. A couple things I had to do to get this working:
- something thinks a filename with dots (e.g. example.8bits.bgen) means there is a python module that needs to be imported (e.g. example), and fails when that does not exist; we have files like this in a data directory
- I could not quickly figure out how to ignore files, so I had to delete a Sphinx conf.py file.
- Sphinx resolves the _templates folder relative to where it is called, not relative to the doc string's source file, so I had to add several symlinks (rather than manually edit every doc string).
---
OK docs are generating. Quite slow, maybe a couple minutes. Looks like one core is at 100% during this process. We've got ~400 files. Maybe its parsing some data files? The whole directory is 318 MB.
---
Now I'm looking at the docs.
First thing I notice is that Python type hint forward references break everything. Consider:
The generated doc page is missing the class name and the entire class definition is inlined as a preformatted block (with highlighting :shrug:).All my python function def's seem to be missing the function name?
No arg functions look really weird
So all our docs are in ReST (thanks Sphinx :|). This means they're not valid Markdown. It seems that invalid markdown in a class doc string can break the formatting of all the methods (presenting them again as one reformatted block).---
On the bright side, the mobile version looks great and the search works better than my experience with Sphinx.
EDIT: formatting
You can manually define the modules for portray (which you probably figured out to get documentation rendering): https://timothycrosley.github.io/portray/docs/quick_start/4....
And, yes write now Markdown only. I will say - this was a week only project: https://timothycrosley.com/project-2-portray so I think there's a good chance I could improve these points with one more week of time spent :)
Thanks!
~Timothy
I've used it for two projects so far: documenting a relatively small python module, and for documenting a RESTful API. I do a lot of embedded work, so I'd like to use it as a place to document a piece of hardware in it's entirety, from schematics/layouts to firmware and build toolchains to actual use of the device. Do you (or anyone else reading this) have any comments on that use case?
I think my main complaint at the moment is that it doesn't play nice with Markdown, so I had to re-format a bunch of pre-existing documentation for it to work in Sphinx. ReST seems to render alright in Gitlab though, so that's a plus.
Haven't tried it yet but you should also be able to enable rst blocks through recommonmark so you get the best of both - https://recommonmark.readthedocs.io/en/latest/auto_structify...
I will give portray a try and see.
Is there any comparison between Sphinx-doc and portray?
[1] http://www.sphinx-doc.org/en/master/
But that's on the devs, not Sphinx, eh?
Enter this command to look for modules in a package, say, foopkg, and create .rst files to generate documentation for each module in the package and subpackages:
I am assuming that the Sphinx documentation project resides in a directory named docs. Each .rst file would contain automodule directives for Sphinx to automatically pull docstrings from foopkg and render them in the generated documentation. However, for Sphinx to understand these automodule directives, the autodoc[2] extension must be enabled in docs/conf.py. If the documentation is written using Google style or NumPy style docstrings, then the napolean[3] extension must also be enabled. The sphinx-apidoc command above would also write a file named docs/api/modules.rst which would be used to render the starting page of the automatically generated API documentation. So include this docs/api/modules.rst with the Sphinx toctree directive from the some other page that the user is likely to visit. For example, the start page of the project documentation would likely be rendered from `docs/index.rst`, so add this to `docs/index.rst`: Now render the documentation with Sphinx normally: [1]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html[2]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html
[3]: https://www.sphinx-doc.org/en/master/usage/extensions/napole...
- Markdown - Zero Config required - Focus on Simplicity
https://timothycrosley.com/project-2-portray goes into why I created the project and how I viewed the current state of Python documentation tool.
Are you planning to support Restructured Text next to Markdown? I find ResT is used more often (in the machine learning ecosystem at least).
> Are you planning to support Restructured Text next to Markdown?
I will look into supporting this, I agree that it definitely makes sense as a feature!