Ensure approved GraphGists all appear individually in search results (p0)

[cheerfulstoic]

Try searching all top search results and making sure they exist in index.
Use Google Webmaster Central to ensure that crawling works well, and there are no errors.

From Ryan:

"I'm looking for attention to the end goal.

If you search for top graphgists in Google, do you find the right results and only the right results? (showing on neo4j.com/graphgist/)

Take a selection of good graphgists and try search.

If you don't find them in the Google index, can you see why? Webmaster central helps with that (and I can give you access). For now, just record the URLs of the GraphGists you're not seeing in the index."

[cheerfulstoic]

I recently filled out the 301s for the node.js portal, so I'm hoping that this will start working better. Currently though as an example if I search for "A March Madness Recommendation Engine" it goes to graphgist.neo4j.com. If people go there it redirects via javascript and google should get a direct 301 when it requests the _escaped_fragment_

[ryguyrg]

Unfortunately Google isn't accepting the 301s to the different site - it's
considering them "soft 404s". I tried Googles "site move" feature and it
doesn't think we're giving 301s even though we are. And a bunch of the
indexed Gists on GraphGist.neo4j.com aren't published now. And redirects
in angular took forever - until the whole page was rendered. So it was a
mess.

I gave up and out in a robots,txt that denies most crawling, except for css
and font and other resources needed for Google to render the pages on
neo4j.com.

I also manually submitted a bunch of the new content on neo4j.com to google
for crawling. Right now if you search google for inurl:neo4j.com/graphgist,
you get s bunch of ugliness.

In particular, lots of Preview titled Gists are indexed. I changed code on
neo4j.com to serve up a meta tag in the header denying robot crawling for
any Gist with a status of "candidate"

On Sunday, February 21, 2016, Brian Underwood [email protected]
wrote:

I recently filled out the 301s for the node.js portal, so I'm hoping that
this will start working better. Currently though as an example if I search
for "A March Madness Recommendation Engine" it goes to graphgist.neo4j.com.
If people go there it redirects via javascript and google should get a
direct 301 when it requests the escaped_fragment

—
Reply to this email directly or view it on GitHub
#64 (comment)
.

ryan boyd
twitter: @ryguyrg http://www.twitter.com/ryguyrg
my book: Getting Started with OAuth 2.0 http://goo.gl/O7K3Y
my website: http://www.developeradvocate.com/

[cheerfulstoic]

Could it not seeing the 301s be because of the _escaped_fragment_? When I was playing with the webmaster tools it seemed to want me to give ?_escaped_fragment_= instead of #!. If I gave #! it just stripped it off.

Good idea about denying crawling to candidate gists. Though if we later mark those as published then could Google refuse to crawl those until we manually go in there? It seems like it would be easy to forget to do that second step

[ryguyrg]

I tried the escaped fragment variants too.

Not sure - spent a few hours of research and trying and then gave up :-(

On candidate Gists, they should never hit Google unless someone publishes a
link to them. Google will also eventually recrawl though I haven't looked
up how long that takes.
On Feb 21, 2016 11:43 PM, "Brian Underwood" [email protected]
wrote:

Could it not seeing the 301s be because of the escaped_fragment? When I
was playing with the webmaster tools it seemed to want me to give
?escaped_fragment= instead of #!. If I gave #! it just stripped it off.

Good idea about denying crawling to candidate gists. Though if we later
mark those as published then could Google refuse to crawl those until we
manually go in there? It seems like it would be easy to forget to do that
second step

—
Reply to this email directly or view it on GitHub
#64 (comment)
.