SAUK Discussion Board

Go Back   SAUK Discussion Board > Board Management > The Garage
Join! Blogs FAQ Calendar Search Today's Posts Mark Forums Read

Notices

Reply  Post New Thread
 
Thread Tools
  #1  
Old 26th January 2019, 17:48
firemonkey firemonkey is offline
Member
 
Join Date: Apr 2010
Location: Calne,Wiltshire
Posts: 5,672

Mood
Balanced

Default ***8217 ***8221

Why are they appearing in posts?
Reply With Quote
  #2  
Old 26th January 2019, 17:51
biscuits biscuits is offline
Member
 
Join Date: Jul 2005
Location: in the tin.
Posts: 18,666
Default Re: ***8217 ***8221

Are you using an iPhone or apple tablet?
Reply With Quote
  #3  
Old 26th January 2019, 17:57
firemonkey firemonkey is offline
Member
 
Join Date: Apr 2010
Location: Calne,Wiltshire
Posts: 5,672

Mood
Balanced

Default Re: ***8217 ***8221

Just a laptop.
Reply With Quote
  #4  
Old 26th January 2019, 18:26
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

People posting with smart quotes turned on in iOS are causing the issue. I've written a fix, just waiting for site admin to apply it.

Are you using a macbook?
Reply With Quote
  #5  
Old 26th January 2019, 21:46
Indigo_ Indigo_ is offline
Co-Administrator
 
Join Date: May 2013
Location: Cheshire
Posts: 19,705
Default Re: ***8217 ***8221

^ I will be passing your comments onto our administrator, who may be able to fix the issue himself
Reply With Quote
  #6  
Old 26th January 2019, 22:37
affluenza affluenza is offline
Banned at own request
 
Join Date: Aug 2013
Location: Working part-time in Retail
Posts: 2,631
Blog Entries: 62
Default Re: ***8217 ***8221

It's because ..8217 and 8217, that wi8221 solve it
Reply With Quote
  #7  
Old 5th February 2019, 10:50
Consolida Consolida is offline
Member
 
Join Date: Dec 2013
Posts: 3,614
Default Re: ***8217 ***8221

This is happening all the time for me. I keep having to edit my posts or they are even more illegible than usual. It’s becoming very tedious
Reply With Quote
  #8  
Old 5th February 2019, 10:56
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

Turn off smart punctuation
Settings>General>Keyboard>Smart Punctuation
Reply With Quote
  #9  
Old 30th July 2019, 18:31
limey123 limey123 is offline
Member
 
Join Date: Feb 2017
Posts: 1,068
Default Re: ***8217 ***8221

^ How do you fix this in Win 10?
Reply With Quote
  #10  
Old 11th August 2019, 13:34
Occultus Occultus is offline
Administrator
 
Join Date: Apr 2006
Location: SAUK HQ
Posts: 18,466

Mood
Inspired

Default Re: ***8217 ***8221

I've made a couple of changes, to see if this issue can be fixed. I suspect I may need to do a couple more things, so if there are any more issues, can you please let me know.

If it does still keep happening for you, it would be helpful if you could send me some details about what device you are using, what you are trying to post, etc., so I can try and replicate the problem at my end.

Please feel free to drop me a PM (or any of the mods) with the details if you encounter any more issues with it
Reply With Quote
  #11  
Old 11th August 2019, 13:48
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

Test:
โ€œIt is absolutely central that parliament should be able to call to account people who represent them as ministers, and at the moment weโ€™re being told by a particular figure, whoโ€™s proud of it, that heโ€™s more or less running the show,โ€

โ–“โ–“โ–“โ–“
โ–’โ–’โ–’โ–“โ–“
โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“โ–“โ–“
โ–’โ–“โ–“โ–“โ–“โ–“โ–“โ–‘โ–‘โ–‘โ–“
โ–’โ–“โ–‘โ–‘โ–‘โ–‘โ–“โ–‘โ–‘โ–‘โ–‘โ–“
โ–“โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–“โ–‘โ–“โ–‘โ–“
โ–“โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–“โ–‘โ–‘โ–‘โ–“
โ–“โ–‘โ–‘โ–“โ–‘โ–‘โ–‘โ–“โ–“โ–“โ–“
โ–’โ–“โ–‘โ–‘โ–‘โ–‘โ–“โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–“โ–“โ–“โ–“โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“โ–“โ–“โ–“
โ–’โ–’โ–’โ–’โ–’โ–“โ–“โ–“โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–’โ–’โ–“โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–’โ–“โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–“โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–“โ–’โ–“โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–“โ–’โ–“โ–“โ–“โ–“โ–“โ–“โ–“โ–“โ–“โ–“
โ–’โ–“โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–“
โ–’โ–’โ–“โ–’โ–’โ–’โ–’โ–’โ–“
Reply With Quote
  #12  
Old 11th August 2019, 14:17
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

Testing testing.

128 โ‚ฌ
20ac โ€š
201a ฦ’
0192 โ€ž
201e โ€ฆ
2026 โ€ 
2020 โ€ก
2021 ห†
02c6 โ€ฐ
2030 ลก
0160 โ€น
2039 ล“
0152 ลพ
144 โ€˜
2018 โ€™
2019 โ€œ
201c โ€
201d โ€ข
2022 โ€“
2013 โ€”
2014 หœ
02dc โ„ข
2122 ลก
0161 โ€บ
203a ล“
0153 ลพ
017e รฟ
Reply With Quote
  #13  
Old 11th August 2019, 14:19
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

Now try editing.

128 โ‚ฌ
20ac โ€š
201a ฦ’
0192 โ€ž
201e โ€ฆ
2026 โ€ 
2020 โ€ก
2021 ห†
02c6 โ€ฐ
2030 ลก
0160 โ€น
2039 ล“
0152 ลพ
144 โ€˜
2018 โ€™
2019 โ€œ
201c โ€
201d โ€ข
2022 โ€“
2013 โ€”
2014 หœ
02dc โ„ข
2122 ลก
0161 โ€บ
203a ล“
0153 ลพ
017e รฟ

ETA: just because.
Reply With Quote
  #14  
Old 11th August 2019, 14:21
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

Yay, we have Unicode back.

โ‰  โ‰ก โ‰ข โ‰ฃ โ‰ค โ‰ฅ โ‰ฆ โ‰ง โ‰จ โ‰ฉ โ‰ช โ‰ซ โ‰ฌ โ‰ญ โ‰ฎ โ‰ฏ โ‰ฐ โ‰ฑ โ‰ฒ โ‰ณ โ‰ด โ‰ต โ‰ถ โ‰ท โ‰ธ โ‰น โ‰บ โ‰ป โ‰ผ โ‰ฝ โ‰พ โ‰ฟ โŠ€ โŠ โŠ‚ โŠƒ โŠ„ โŠ… โŠ† โŠ‡ โŠˆ โŠ‰ โŠŠ โŠ‹ โŠŒ โŠ โŠŽ โŠ

ETA: And we have changed to UTF-8
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
Reply With Quote
  #15  
Old 11th August 2019, 14:53
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

The punctuation that did get through to the database before now doesn't render. I guess it is not an issue though.

^^^^ That is probably why Unicode was blocked in the first place.
Reply With Quote
  #16  
Old 11th August 2019, 14:57
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

Yeah it's odd its not rendering, they only needed to replace *** with &# I would have thought.
Reply With Quote
  #17  
Old 11th August 2019, 14:58
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

ohhhhh &# is still in the swear filter.
Reply With Quote
  #18  
Old 11th August 2019, 18:38
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

^^ I think that the fact that they don't render is how it should be if the database is ISO-8859-1. They are not printable characters in that encoding.

They shouldn't have made it in in the first place and should have been turned to Unicode and then redacted.

They are held natively so there is no *** to replace.

Also just doing a replace on *** might make more of a mess than expected. The swear filter is probably lossy and there will be no way to tell if the *** is there to replace the word ***.

Maybe the punctuation that did get through could be converted back if it still exists natively in the database.

The fact that the swear filter has not been changed does hint at a change of coding to UTF-8 in the database? The database could be holding the Unicode natively so it doesn't have to use the naughty prefix. That may explain why I am seeing the same unprintable character for different punctuation. I will double check what I am getting.

This solution to the issue makes it even more unclear why the prefix is being redacted if was not to simply disable Unicode.
Reply With Quote
  #19  
Old 11th August 2019, 18:41
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

No, they are held in the database with the stars.

It would be quite easy to replace using regex.

I'll test the changes on my copy and see what happens.
Reply With Quote
  #20  
Old 11th August 2019, 18:42
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

Thanks Occultus for all your hard work behind the scenes.
Reply With Quote
  #21  
Old 11th August 2019, 18:45
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

^^ Not the few extended punctuation characters that were getting trashed after editing. They must have been getting through natively.

See my post:-

Quote:
Originally Posted by Sisyphus
The reson I was think what I did was that ISO-8859-1 doesn't contain the extended punctuation and yet they are still stored without being converted to Unicode. It is windows-1252 that does contain them.

If the board strictly used ISO-8859-1 throughout ๏ฟฝ ๏ฟฝ ๏ฟฝ ๏ฟฝ ๏ฟฝ ๏ฟฝ ๏ฟฝ ๏ฟฝ would never get to the database and yet here they are in the database.

As I said, I could be wrong.
Reply With Quote
  #22  
Old 11th August 2019, 18:46
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

Oh maybe when using Go Advanced. ะ‚
Reply With Quote
  #23  
Old 11th August 2019, 18:52
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

Do you think the database has been changed to UTF-8?

How else could all this Unicode bypass the swear filter?
Reply With Quote
  #24  
Old 11th August 2019, 18:54
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

I just changed my database to UTF-8 and the forum and I get the same bad characters.

It's still stored correctly in the database.
Reply With Quote
  #25  
Old 11th August 2019, 18:57
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

OK, I've found the issue. It needs changing in a config file too.

I'll message him.
Reply With Quote
  #26  
Old 13th August 2019, 04:44
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

I donโ€™t think the loss of the former punctuation marks is due to settings. My guess would be that the database has been converted to UTF-8 from ISO-8859-1 rather than from windows-1252.

The extended punctuation marks that formerly got into the database were windows-1252. Whatever software transcoded the database didnโ€™t know what to do with the unprintable (in ISO-8859-1) characters and replaced them with Unicode FFFD.

Under HTML5 I think the transcoder โ€˜shouldโ€™ have assumed windows-1252 for anything it didnโ€™t recognize and if it had it would have got it right.

This makes me think there may be a mix of HTML and HTML5 compatible code in the engine room and may be how the extended punctuation got into the database in the first place.

If this is what has happened then some information may have been lost as all of the extended punctuation is now FFFD. Some effort could be made to mitigate this but I guess it will soon fade into the past and can sit alongside the ***8221; as a curiosity.
Reply With Quote
  #27  
Old 13th August 2019, 04:47
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

No I tested it on my install, changing the setting made it display correctly.
Reply With Quote
  #28  
Old 13th August 2019, 04:52
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

Then I am missing something.

What setting did you change? What does it do?
Reply With Quote
  #29  
Old 13th August 2019, 05:06
Sisyphus Sisyphus is offline
Member
 
Join Date: Aug 2012
Location: Lost
Posts: 375
Blog Entries: 1
Default Re: ***8217 ***8221

Unless it is transcoding on the fly then it can't be corrected by changng a setting now.

If it is transcoding on the fly then that implies the database is still in the old format.

If that is the case then the database cannot hold Unicode because the swear filter will stomp on it.



To see what I am seeing I think the database must now be UTF-8 and that must have happened at one epoch. That being the case then the information has now been lost.

But I have been wrong before.
Reply With Quote
  #30  
Old 13th August 2019, 05:22
Schmosby Schmosby is online now
Member
 
Join Date: Jan 2012
Location: London
Posts: 4,366
Blog Entries: 1

Mood
Relaxed

Default Re: ***8217 ***8221

I think the setting changes the way mysqli handles the data.

Code:
    //    ****** MySQLI OPTIONS ***** 
    // When using MySQL 4.1+, MySQLi should be used to connect to the database.
    // If you need to set the default connection charset because your database
    // is using a charset other than latin1, you can set the charset here.
    // If you don't set the charset to be the same as your database, you
    // may receive collation errors.  Ignore this setting unless you
    // are sure you need to use it. 
// $config['Mysqli']['charset'] = 'utf8';
The data is stored correctly in the database, it's only on the site it shows as question marks. Once the setting is changed the unicide will show correctly.
Reply With Quote
Reply

Thread Tools

Forum Jump


All times are GMT +1. The time now is 14:14.


SAUK Award
Logo designed by abc
Powered by vBulletin
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.