Walkthrough

How to Remove Duplicates items in SharePoint Online List and keep one by PowerShell ?

Posted on Updated on

Hello folks,

We have faced recently a critical situation with a customer:
He was running a certain PowerShell script to update a SharePoint Online list with Data, it reaches 7000 items, a certain issue occurred in the script, and he found that he as 20,000 items in the same list, 70% was duplicates.

He wants to solve the situation in a safe way with no risk, by removing duplicate items and keep the Original one.
To solve the situation I did the following, This solution can be applied on all SharePoint versions 2010, 2013, 2016, & SharePoint Online, as the script is written to work as Client Object Model.

      1. I took back-up from the List, if you use SharePoint On-premise version, you can take back-up of the site. If you are using SharePoint Online, you can use Sharegate, however this step is not mandatory.
      2. Export AllItems to Excel using the Excel action in the ribbon, and make sure the ID Column is part of the export.
      3.  Download and setup a free tool named Kutools for Excel from here , the page contains also description how to use it to mark the duplicated column.
      4. Now take of the IDs from the Excel, and paste it in a text file on your harddrive
        Ids
      5. Then run the below PowerShell Script
        Add-Type -Path "C:\Path\SPOnline\Microsoft.SharePoint.Client.dll"
        Add-Type -Path "C:\Path\SPOnline\Microsoft.SharePoint.Client.Runtime.dll"
        $siteurl = "https://tenant.sharepoint.com/sites/yourSite/"
        $UserName = "siteadmin@tenant.onmicrosoft.com"
        $SecurePassword = Read-Host -Prompt "Please enter your password" -AsSecureString
        $Credentials = New-Object -TypeName System.Management.Automation.PSCredential -argumentlist $userName, $SecurePassword
        $SPOCredentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $SecurePassword)
        $context = New-Object Microsoft.SharePoint.Client.ClientContext($siteurl)
        $context.Credentials = $SPOCredentials
        $web = $context.Site.RootWeb
        $context.Load($web)
        $context.Load($web.Lists)
        $context.ExecuteQuery()
        $listTitle = "List Name"
        $list = $Context.Web.Lists.GetByTitle($listTitle)
        $Context.Load($list)
        $Context.ExecuteQuery()
        $TargetSites = Get-Content "C:\temp\List-of-Ids.txt"
        foreach ($targetSite in $TargetSites)
        {
        $id=$targetSite.Split(";")[0]
        $caml = @"<View Scope="RecursiveAll"><Query><OrderBy><FieldRef Name='ID' Ascending='TRUE'/></OrderBy><Where><Eq><FieldRef Name='ID' /><Value Type='Number'>$id</Value></Eq></Where></Query><RowLimit Paged="TRUE">1</RowLimit></View>"
        $query = new-object Microsoft.SharePoint.Client.CamlQuery
        $query.ViewXml = $caml
        $items = $list.GetItems($query)
        $Context.Load($items)
        $Context.ExecuteQuery()
        if ($items.Count -gt 0)
        {
        for ($i = $items.Count-1; $i -ge 0; $i--)
        {
        Write-Host "Deleted: " $items[$i].Id.ToString() -ForegroundColor Yellow
        }
        $Context.ExecuteQuery()
        }
        }
        #$context.Dispose()

     

    Now, the items will start to be deleted, you can change the PowerShell script to fits your need.
    This method is safe because you see exactly what do you want to delete before executing the deletion script

Advertisements

Get SharePoint Online Event Receivers

Posted on Updated on

In this article we will get list of Event receivers attached to SiteCollection or Web Levels.
Please follow these steps :

  1. Make sure Microsoft.SharePoint.Client.dll and Microsoft.SharePoint.Client.Runtime.dll
  2. Open PowerShell ISE & Run the Below script after setting the target parameter
    # Paths to SDK. Please verify location on your computer.
    Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
    Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
    #declare the variables
    $tenantSite = "https://tenant-admin.sharepoint.com"
    #read file
    $siteurl = "https://tenant.sharepoint.com/teams/site1"
    $UserName = "admin@tenant.onmicrosoft.com"
    $SecurePassword = Read-Host -Prompt "Please enter your password" -AsSecureString
    $Credentials = New-Object -TypeName System.Management.Automation.PSCredential -argumentlist $userName, $SecurePassword
    $SPOCredentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $SecurePassword)
    Connect-SPOService -Url $tenantSite -Credential $Credentials
    $context = New-Object Microsoft.SharePoint.Client.ClientContext($siteurl)
    $context.Credentials = $SPOCredentials
    $web = $context.Site.RootWeb #You can make it only $context.Site if you want to get the Events on Site Collection level
    $context.Load($web)
    $context.Load($web.EventReceivers)
    $context.ExecuteQuery()
    $eventReceivers = $web.EventReceivers
    foreach($eventReceiver in $eventReceivers)
    {
    Write-Host $eventReceiver.ReceiverId " " $eventReceiver.EventType.ToString() " " $eventReceiver.ReceiverName
    }
  3. Output will be as follow

Of course the above script can be modified to be compatible with If you want to verify your results, you can use SharePoint Client Browser (SPCB)
It is a free helpful open source tool that can give you nice insights about your Site/Web event receivers and it supports 3 versions (SP2013, SP2016, & SPOnline)

 

Get SharePoint Admins for All SharePoint Online Sites

Posted on Updated on

In this article we will get list of all SharePoint Online Sites, and then List Site Admins using PnP library
Please follow these steps :

      1. Set-up PnP module latest version from here https://github.com/SharePoint/PnP-PowerShell/releases
      2. Run the Below script after setting the target parameter
        #Connect to SPO tenant
        $CurrentCred = Get-Credential
        Connect-SPOService "https://admin-admin.sharepoint.com" -Credential $CurrentCred
        #Get all Site collections
        $sites = Get-SPOSite -Limit All
        foreach ($site in $sites)
        {
        Connect-PnPOnline -Url $site.Url -Credential $CurrentCred
        $admins = Get-PnPSiteCollectionAdmin | select Title #You can also add ,Email
        $allAdmins=""
        foreach($admin in $admins)
        {
        $allAdmins += $admin.Title +";" #You can have also $admin.Email
        }
        Write-host ($site.Url+","+$allAdmins) -ForegroundColor Green
        ($site.Url+","+$allAdmins) >> "C:\temp\SiteCollectionAdmins.csv"
        }

Then you will have a CSV file with SiteUrl & Site Admin Name delemited by ;

Apply SharePoint Online template using PnP PowerShell

Posted on

In this article we will define a template, and apply it for other SPO sites using PnP library
Please follow these steps :

    1. Set-up PnP module latest version from here https://github.com/SharePoint/PnP-PowerShell/releases
    2. Go to your SPO tenant, and create a SPO Site, you can also define (List structure, Theme, Pages, Content Types) because all of those will be part of the template.
    3. Make sure that the app catalog is activated on the tenant (There is no relation between app catalog and the tenant, but there is a bug found in January PnP package that required App catalog to be activated or a failure occurred)
    4. After having the site ready, run the below PowerShell command:
      $Creds = Get-Credential
      $SiteURL = "https://test.sharepoint.com/sites/demo1/"
      Connect-PnPOnline -Url $SiteURL -Credentials $Creds
      Get-PnPProvisioningTemplate -Out C:\temp\demo1Template.xml
    5. You should see a progress running for a while to export all settings.
    6. Please open the XML file at the provisioned path (C:\temp\demo1Template.xml), and have a look, it is also doable to do medications through the file, by adding items, or fields, but you need to be careful not to crap the schema.
    7. Now it is time to import it to the destination site, Create a destination site on SPO with a default teamsite template
    8. Apply the new template by running the below command.
      $DestUrl = "https://test.sharepoint.com/sites/destination1/"
      Connect-PnPOnline -Url $DestUrl -Credentials $Creds
      $template = Load-PnPProvisioningTemplate -Path "C:\temp\demo1Template.xml"
      Apply-PnPProvisioningTemplate -InputInstance $template

After this, refresh the destination site, and you should see the template applied.
I suggest to start first with a simple template, then apply more complexity to it. this approach should run smoothly also with SP2016, but I haven’t tried to import a template from SPO and import it to 2016 or vise versa

“Access Denied” to Access Requests list or “Request approval failed” when you process a pending request in SharePoint Online

Posted on Updated on

I have run through a problem today during my work on one of the clients, and this sometimes happened mainly with sites that is migrated to SharePoint Online

 

 

 

This works perfectly fine !

Draw PowerBI Map from IP Addresses

Posted on

I had a request from a client, where he has an Excel sheet with list of IP addresses as well as other information, and he want to convert those IP address to the equivalent country and list in PowerBI.

Some Analytical tool like Splunk have this is a built in function, but for PowerBI you need to build it your self, and we will rely on a free web-service to implement this, so lets start

  1. Go to http://freegeoip.net/json/
  2. You should see the default data, which is your data retrieved by your IP address.
  3. Data displayed will be Country Code, Country Name, Region Code, Region Name, City, Zip Code(if applicable), Time Zone, Latitude, Longitude, and Metro Code
  4. if you entered specific IP address, correspondence data will be displayed, example: http://freegeoip.net/json/148.50.4.18
  5. Now, lets get back to PowerBI, Open your , Click Get Data, and then select Blank Query
  6. Click View, then Advanced Query, and add the Query in the right text
  7. Copy paste this Query and enter it in the text box of the Advanced Query
    let
    Source = (#"IP Address" as text) => let
    Source = Json.Document(Web.Contents("http://freegeoip.net/json/" & #"IP Address")),
    #"Converted to Table" = Record.ToTable(Source),
    #"Transposed Table" = Table.Transpose(#"Converted to Table"),
    #"Promoted Headers" = Table.PromoteHeaders(#"Transposed Table")
    in
    #"Promoted Headers"
    in
    Source
  8. Click Done, and rename the function with a meaningful name like : fn_GetRegion
  9. Now, you need to create a column with this function, to do so, from Edit Queries go to Add Column > Invoke Custom Function
  10. Enter the data as shown below
  11. The column will appear at the end of your table, now we need to Expand the column by click on the double arrow icons as in the snapshot below
  12. Now, all the columns is appeared clearly.
  13. and you can use it to draw your maps.

Thanks to guavaq who shows us the solution in this post

Let’s follow protocols: HTTP

Posted on Updated on

Some of us know a lot about what happen when we write URL in our browser and hit enter or when an app connect to a web service or API to get data or perform an operation, but let us discuss exactly what is going on between the browser or any client and the URL  or any server resource like HTML page, files, or web service … etc. As many knows HTTP sort for Hypertext Transfers Protocol so every thing happen is written as text on this dialog between the client and the server but what is exactly this message is and what is the server replay that what we will try to clarify.

First let us discuss the client which may be web browser, app, or even an web debugging tools like fiddler, Post Man, SOAP UI. which I think we need to talk about after HTTP.

Client Message or HTTP Request, in a happy world the request between client and server simply done by sending the request with a certain structure (we will discuss it in details) to the server and wait for the server response, but this request can go throw a long way by passing to proxy, gateway, and/or tunnel. In sort, A proxy is a forwarding agent, receiving requests for a URL in its absolute form, rewriting all or part of the message, and forwarding the reformatted request toward the server identified by the URL. A gateway is a receiving agent, acting as a layer above some other server(s) and, if necessary, translating the requests to the underlying server’s protocol. A tunnel acts as a relay point between two connections without changing the messages; tunnels are used when the communication needs to pass through an intermediary (such as a firewall) even when the intermediary cannot understand the contents of the messages.

 

 

HTTP Request message structure (Header and body):

1- Request Line contains [Request Method] + [Server Resource URL] + [HTTP Protocol version] + [Carriage return] + [Line feed]

ex: GET /pdf/book.pdf HTTP/1.1

2- Request header fields list, all header fields are optional except the host field. and the field structure is [Field Name] + [Colon] + [Field Value] + [Carriage return] + [Line feed]

ex: Host: tecgang.wordpress.com

3-Empty Line  [Carriage return] + [Line feed] in order to split between request header and request body

4-[Optional] Request body, which can be anything, binary file, text, JSON data or any data you want to send it to the server

HTTP Request methods are:

GET: The GET method requests a representation of the specified resource. for example to get an image or html file.

HEAD: The HEAD method asks for a response identical to that of a GET request, but without the response body. for example if you want to get a file but you want to know the file length first then you request a HEAD. (Useful for retrieving meta-information)

POST: Used to ask the server to accept the entity enclosed in the request as a new entity of the resource identified by the URL.

PUT: Used to ask the server to accept the entity enclosed in the request to store an entity of the resource identified by the URL. if it not exists its create new one or update existing one.

 DELETE: The DELETE method deletes the specified resource.

TRACE: The TRACE method echoes the received request so that a client can see what (if any) changes or additions have been made by intermediate servers.

OPTIONS: The OPTIONS method returns the HTTP methods that the server supports for the specified URL.

CONNECT: The client asks an HTTP Proxy server to tunnel the TCP connection to the desired destination. The server then proceeds to make the connection on behalf of the client. Once the connection has been established by the server, the Proxy server continues to proxy the TCP stream to and from the client.

PATCH: The PATCH method update partial resources.

 

 

HTTP Response message structure (Header and body):

1- Status Code and message [HTTP Version] + [Status Code] + [Message] + [Carriage return] + [Line feed]

ex: HTTP/1.1 200 OK

2- Response header fields, all header fields are optional and the field structure is [Field Name] + [Colon] + [Field Value] + [Carriage return] + [Line feed]

ex: Content-Type: application/pdf

3- Empty Line  [Carriage return] + [Line feed] in order to split between response header and response body

4-[Optional] Request body, which can be anything, binary file, text, JSON data or any data send by the server

 

 

Now you can see whats happen if you type in your browser an url like this one https://tecgang.wordpress.com/index.php, the dialog will be:

The Request:

GET /index.php HTTP/1.1
Host: tecgang.wordpress.com

The Response:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: 138

<html>
<head>
<title>An Example Page</title>
</head>
<body>
Hello World, this is a very simple HTML document.
</body>
</html>